1. Field of the Invention
Embodiments of the invention provide techniques for generating Extract, Transform, and Load (ETL) jobs. More specifically, embodiments of the invention relate to techniques for generating ETL jobs for loading data incrementally.
2. Description of the Related Art
ETL jobs typically integrate data from multiple, heterogeneous data sources into a central repository such as a data warehouse. Integrated data in the central repository may need to be periodically refreshed to prevent the integrated data from turning stale (due to data being changed at one of the data sources, for example). However, periodically reloading all of the data from the data sources may be unacceptably costly in terms of performance. Instead, the data may be loaded from the data sources incrementally. Incrementally loading data refers to a technique of capturing changes at the data source and propagating the changes to the central repository. For example, an ETL job may infer, from changes captured at the data source, how to update the central repository accordingly. However, it may be costly to have an ETL programmer create each such ETL job for incrementally loading data.